Papers with dynamic evaluation framework
A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the player experience in a roleplaying game augmented with LLM-generated dialogue remains a major challenge. |
| Approach: | They propose a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games. |
| Outcome: | The proposed framework directly evaluates the performance of LLM-generated dialogue in a role-playing game with 28 players. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |
Dynamic Evaluation with Cognitive Reasoning for Multi-turn Safety of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing safety evaluation methods rely on static assessments that use fixed harmful prompts or predefined prefixes as jailbreak templates. |
| Approach: | They propose a dynamic evaluation framework for multi-turn safety assessment of LLMs based on cognitive theories to simulate real chatting process and scenario simulation and strategy decision to guide dynamic generation. |
| Outcome: | The proposed framework has been applied to evaluate the safety of widely used LLMs. |